Probabilistic Linkage of Persian Record with Missing Data

Extended Abstract. When the comprehensive information about a topic is scattered among two or more data sets, using only one of those data sets would lead to information loss available in other data sets. Hence, it is necessary to integrate scattered information to a comprehensive unique data set. On the other hand, sometimes we are interested in recognition of duplications in a data set. The identification of duplications in a data set or the same identities in different data sets is called record linkage. Linkage of data sets that their information is registered in the context of Persian language has special difficulties due to particular writing characteristics of the Persian language such as connectedness of letters in words, existence of different writing versions for some letters and dependency of writing shape of letters to their position in words. In this paper, usual difficulties in linkage of data sets that their information is registered in the context of the Persian language are studied and some solutions are presented. We introduced some compatible methods for preparing and preprocessing of files through standardization, blocking and selection of identifier variables. A new method is proposed for dealing with missing data that is a major problem in real world applications of record linkage theory. The proposed method takes into account the probability of occurrence of missing data. We also proposed an algorithm for increasing the number of comparable fields based on partitioning of composite fields such as address. Finally, the proposed methods are used to link records of establishing censuses in a geographical region in Iran. The results show that taking into account the probability of the occurrence of missing data increases the efficiency of the record linkage process. In addition, using different codes and notations for data registration in different times, leads to information loss. Specially, it is necessary to design a general pattern for writing addresses in Iran, considering geographical and environmental situations.